Efficient Hold-Out for Subset of Regressors

Hold-out and cross-validation are among the most useful methods for model selection and performance assessment of machine learning algorithms. In this paper, we present a computationally efficient algorithm for calculating the hold-out performance for sparse regularized least-squares (RLS) in case the method is already trained with the whole training set. The computational complexity of performing the hold-out is O(|H |3 + |H |2n), where |H | is the size of the hold-out set and n is the number of basis vectors. The algorithm can thus be used to calculate various types of cross-validation estimates effectively. For example, when m is the number of training examples, the complexities of N-fold and leave-one-out cross-validations are O(m/N + (mn)/N) and O(mn), respectively. Further, since sparse RLS can be trained in O(mn) time for several regularization parameter values in parallel, the fast hold-out algorithm enables efficient selection of the optimal parameter value.